Systems and methods for processing electronic images to identify transplant donor-recipient matches

ABSTRACT

Systems and methods are described herein for processing electronic medical images to predict one or more donor recipients for a patient. For example, a digital medical image of the patient may be received, wherein the patient is in need of a transplant. A trained machine learning system may be determined. The digital medical image may be provided into the trained machine learning system, the trained machine learning system determining a patient embedding. Using the patient embedding, a subset of donor recipients may be determined. Based on the subset of donor recipients a recommendation of optimal donors may be determined.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit of priority to U.S. ProvisionalPatent Application No. 63/366,015, filed Jun. 8, 2022, the entirety ofwhich is incorporated by reference herein.

FIELD OF THE DISCLOSURE

Various embodiments of the present disclosure pertain generally toprocessing electronic images. More specifically, particular embodimentsof the present disclosure relate to systems and methods for processingelectronic images, among other metadata, using artificial intelligence(AI) technology, machine learning, and/or image processing techniques toidentity transplant donor-recipient matches.

BACKGROUND

Many diseases are treated by transplanting organs, tissues, or otherforeign materials into a patient from a donor. As one example, a patienthaving a faulty organ due to disease, e.g., a faulty lung or heart dueto lung or heart failure, may undergo an organ transplant to replace thefaulty organ. As another example, a patient needing treatment for agastrointestinal disease, such as recurrent Clostridium difficilecolitis, may undergo a fecal microbiota transplantation (FMT), alsoreferred to as bacteriotherapy, where stool from a healthy donor istransferred into the gastrointestinal tract of the patient.Additionally, ongoing studies are investigating bacteriotherapy'sefficacy for treating other diseases, such as obesity, overall health,cancer, and non-alcoholic fatty liver disease. However, identifying theoptimal donor for treating the patient's disease and ensuring that thepatient benefits is challenging, and a failure to do so my lead tosevere, life-threatening consequences for the patient.

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Unless otherwiseindicated herein, the materials described in this section are not priorart to the claims in this application and are not admitted to be priorart, or suggestions of the prior art, by inclusion in this section.

SUMMARY OF THE DISCLOSURE

According to certain aspects of the present disclosure, systems andmethods are disclosed for processing electronic medical images. In oneaspect, a computer-implemented method for processing electronic medicalimages to predict one or more donor recipients for a patient. The methodmay comprise: receiving a digital medical image of the patient, whereinthe patient is in need of a transplant; determining a trained machinelearning system; providing the digital medical image into the trainedmachine learning system, the trained machine learning system determininga patient embedding; determining, using the patient embedding, a subsetof donor recipients; and determining based on the subset of donorrecipients a recommendation of optimal donors.

The transplant may be a fecal matter transplant. The transplant may be aliver transplant. A salient region detection module may be applied todetermine a saliency of each region within the received digital medicalimage, and non-salient image regions are excluded from processing by thetrained machine learning system. Metadata associated with the patientmay be received, the metadata including: clinical data, geneticinformation, microbial composition, and/or life history data; and themetadata may be provided into the trained machine learning system.

A second trained machine learning system may be determined, the secondtrained machine learning system being capable of determining a dietary,sleep, or exercise suggestion, wherein the second trained machinelearning system receives as input the digital medical image of thepatient, determines a lifestyle embedding, and determines, based on thelifestyle embedding, a dietary, sleep, or exercise suggestion for thepatient.

Donor collection entities may be notified, with a notification, of aneed for donors with a given profile. The notification may includemetadata of the patient in need, an indication of the donor profileneeded that matches the recipient, and/or a request for the donorcollection entities to start collecting donors that have a similarprofile.

According to certain aspects of the present disclosure, systems andmethods are disclosed for processing electronic medical images. Inanother aspect, a system for processing electronic digital medicalimages may comprise at least one memory storing instructions and atleast one processor configured to execute the instructions to performoperations. The system for processing electronic digital medical imagesmay predict one or more donor recipients for a patient. The at least oneprocessor may comprise: receiving a digital medical image of thepatient, wherein the patient is in need of a transplant; determining atrained machine learning system; providing the digital medical imageinto the trained machine learning system, the trained machine learningsystem determining a patient embedding; determining, using the patientembedding, a subset of donor recipients; and determining based on thesubset of donor recipients a recommendation of optimal donors.

According to certain aspects of the present disclosure, systems andmethods are disclosed for processing electronic medical images. Inanother aspect, a non-transitory computer-readable medium storinginstructions that, when executed by a processor, perform operationsprocessing electronic digital medical images, is disclosed. Theoperations may predict one or more donor recipients for a patient. Theoperations may comprise: receiving a digital medical image of thepatient, wherein the patient is in need of a transplant; determining atrained machine learning system; providing the digital medical imageinto the trained machine learning system, the trained machine learningsystem determining a patient embedding; determining, using the patientembedding, a subset of donor recipients; and determining based on thesubset of donor recipients a recommendation of optimal donors.

Additional objects and advantages of the disclosed embodiments will beset forth in part in the description that follows, and in part will beapparent from the description, or may be learned by practice of thedisclosed embodiments. The objects and advantages of the disclosedembodiments will be realized and attained by means of the elements andcombinations particularly pointed out in the appended claims. As will beapparent from the embodiments below, an advantage to the disclosedsystems and methods is that multiple parties may fully utilize theirdata without allowing others to have direct access to raw data. Thedisclosed systems and methods discussed below may allow advertisers tounderstand users' online behaviors through the indirect use of raw dataand may maintain privacy of the users and the data.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1A illustrates an exemplary block diagram of a system and networkfor processing images to determine a donor recipient prediction,according to techniques presented herein.

FIG. 1B illustrates an exemplary block diagram of a tissue viewingplatform according to techniques presented herein.

FIG. 1C illustrates an exemplary block diagram of a slide analysis tool,according to techniques presented herein.

FIG. 2 illustrates an exemplary process for to determining a donorrecipient prediction, according to techniques presented herein.

FIG. 3A is a flowchart illustrating an example method of training analgorithm for region detection, according to an exemplary embodiment ofthe present disclosure.

FIG. 3B is a flowchart illustrating an exemplary method of utilizing analgorithm for region detection, according to an exemplary embodiment ofthe present disclosure.

FIG. 4A is a flowchart illustrating an example method of training adonor recipient prediction module according to an exemplary embodimentof the present disclosure.

FIG. 4B is a flowchart illustrating an exemplary method of utilizing adonor recipient prediction according to an exemplary embodiment of thepresent disclosure.

FIG. 5A is a flowchart illustrating an example method of training adietary, lifestyle, and lifespan recommendation module according to anexemplary embodiment of the present disclosure.

FIG. 5B is a flowchart illustrating an exemplary method of utilizing adietary, lifestyle, and lifespan recommendation module according to anexemplary embodiment of the present disclosure.

FIG. 6 illustrates an exemplary process for determining a FMT donorrecipient, according to techniques presented herein.

FIG. 7 illustrates an exemplary process for to determining a dietary,lifestyle and lifespan recommendations, and/or a delivery typerecommendation, according to techniques presented herein.

FIG. 8 illustrates an exemplary flowchart for processing images todetermine a donor recipient prediction, according to techniquespresented herein.

FIG. 9 depicts an example of a computing device that may executetechniques presented herein, according to one or more embodiments.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the exemplary embodiments of thepresent disclosure, examples of which are illustrated in theaccompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.

The systems, devices, and methods disclosed herein are described indetail by way of examples and with reference to the figures. Theexamples discussed herein are examples only and are provided to assistin the explanation of the apparatuses, devices, systems, and methodsdescribed herein. None of the features or components shown in thedrawings or discussed below should be taken as mandatory for anyspecific implementation of any of these devices, systems, or methodsunless specifically designated as mandatory.

Also, for any methods described, regardless of whether the method isdescribed in conjunction with a flow diagram, it should be understoodthat unless otherwise specified or required by context, any explicit orimplicit ordering of steps performed in the execution of a method doesnot imply that those steps must be performed in the order presented butinstead may be performed in a different order or in parallel.

As used herein, the term “exemplary” is used in the sense of “example,”rather than “ideal.” Moreover, the terms “a” and “an” herein do notdenote a limitation of quantity, but rather denote the presence of oneor more of the referenced items.

Transplants of tissue, organs, or foreign matter into a patient fortreating disease is common. Exemplary forms of transplantation fromdonors to patients include: microbial bioflora transplantation, e.g.,fecal microbiota transplantation (FMT), which is done by having banks offecal samples from donors; organ transplantation from living donors,which occurs when an organ can be removed from a donor without death,e.g., removal of a kidney or a portion of the liver; organtransplantation from deceased or brain-dead donors, which includescritical organs such as the heart and lungs that are viable for about 24hours after the donor's death; and tissue transplantation from adeceased or brain-dead donor, including bones, skin, heart valves,nerves, corneas, and veins, which may be preserved in a tissue bank forup to five years. In some examples, transplant donors may also includeanimals, such as pigs.

Although transplants of tissues, organs, or foreign matter into apatient are common, identifying donor tissues, organs, or foreign matterthat lead to a best outcome is challenging. Techniques discussed hereinmay use AI technology, machine learning, and/or image processing toolsapplied to patient and donor data, e.g., digital images of histologyslides, radiology, clinical reports, genetic information, microbialcomposition, life history, etc., to identify optimal donors for apatient in need of a transplant. These techniques include, but are notlimited to, identifying optimal donors for FMT and identifying optimaldonors for organ transplantation. In some examples, the identificationof optimal donors for the patient may be one filter (e.g., one datapoint) implemented by a medical professional when determining candidatedonors for a transplant that may, e.g., increase a confidence of themedical professional in selecting a most optimal candidate donor thatwill provide a best outcome for the patient. Additional techniquesdiscussed herein may use AI technology, machine learning, and imageprocessing tools to provide dietary, lifestyle, and/or lifespanrecommendations to improve a transplant patient's outcome ad/or tofurther optimize the donor-patient match. Further techniques discussedherein may automatically notify donor collection entities of donor typesneeded for (e.g., to match) transplant patients.

FIG. 1A illustrates a block diagram of a system and network forprocessing images to determine a determine a donor recipient prediction,according to an exemplary embodiment of the present disclosure.

Specifically, FIG. 1A illustrates an electronic network 120 that may beconnected to servers at hospitals, laboratories, and/or doctors'offices, etc. For example, physician servers 121, hospital servers 122,clinical trial servers 123, research lab servers 124, and/or laboratoryinformation systems 125, etc., may each be connected to an electronicnetwork 120, such as the Internet, through one or more computers,servers, and/or handheld mobile devices. According to an exemplaryembodiment of the present disclosure, the electronic network 120 mayalso be connected to server systems 110, which may include processingdevices 111 that are configured to implement a tissue viewing platform100, which includes a slide analysis tool 101 for determining specimenproperty or image property information pertaining to digital pathologyimage(s), and using machine learning to classify a specimen, accordingto an exemplary embodiment of the present disclosure. The tissue viewingplatform 100 may also include a donor recipient inference tool 141 fordetermining a donor recipient prediction. In other examples, the donorrecipient inference tool 141 may be operated separately from (e.g., by adifferent platform than) the tissue viewing platform 100. The tissueviewing platform 100 may also include a dietary and lifestyle tool 144for determining a dietary, lifestyle, and/or lifespan recommendation. Inother examples, the dietary and lifestyle tool 144 may be operatedseparately from (e.g., by a different platform than) the tissue viewingplatform 100.

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124, and/or laboratory information systems 125may create or otherwise obtain images of one or more patients' cytologyspecimen(s), histopathology specimen(s), slide(s) of the cytologyspecimen(s), digitized images of the slide(s) of the histopathologyspecimen(s), or any combination thereof. The physician servers 121,hospital servers 122, clinical trial servers 123, research lab servers124, and/or laboratory information systems 125 may also obtain anycombination of patient-specific information, such as age, medicalhistory, cancer treatment history, family history, past biopsy orcytology information, etc. The physician servers 121, hospital servers122, clinical trial servers 123, research lab servers 124, and/orlaboratory information systems 125 may transmit digitized slide imagesand/or patient-specific information to server systems 110 over theelectronic network 120. Server systems 110 may include one or morestorage devices 109 for storing images and data received from at leastone of the physician servers 121, hospital servers 122, clinical trialservers 123, research lab servers 124, and/or laboratory informationsystems 125. Server systems 110 may also include processing devices forprocessing images and data stored in the one or more storage devices109. Server systems 110 may further include one or more machine learningtool(s) or capabilities. For example, the processing devices may includea machine learning tool for a tissue viewing platform 100, according toone embodiment. Alternatively or in addition, the present disclosure (orportions of the system and methods of the present disclosure) may beperformed on a local processing device (e.g., a laptop).

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124, and/or laboratory information systems 125refer to systems used by pathologists for reviewing the images of theslides. In hospital settings, tissue type information may be stored inone of the laboratory information systems 125.

FIG. 1B illustrates an exemplary block diagram of the tissue viewingplatform 100. For example, the tissue viewing platform 100 may include aslide analysis tool 101, a donor recipient inference tool 141, a dietaryand lifestyle tool 144, a data ingestion tool 102, a slide intake tool103, a slide scanner 104, a slide manager 105, a storage 106, and aviewing application tool 108.

The slide analysis tool 101, as described below, refers to a process andsystem for processing digital images associated with a tissue specimen(e.g., digitized images of slide-mounted histology or cytologyspecimens), and using machine learning to analyze a slide, according toan exemplary embodiment.

The donor recipient inference tool 141, as described in greater detailbelow, refers to a process and system for processing digital pathologyslides (e.g., digitalized images of a slide-mounted history or cytologyspecimens) and/or metadata, and using machine learning or a rules basedsystem for determining a patient embedding, a subset of recipients, andan optimal donor prediction. The trained system may have two parts, aembedding tool 142 and a donor recommendation tool 143, embedding tool142 determining patient embeddings, and the donor recommendation tool143 may determine one or more donor recommendation based on thedetermined patient embedding.

The dietary and lifestyle tool 144, as described in greater detailbelow, refers to a process and system for processing digital pathologyslides (e.g., digitalized images of a slide-mounted history or cytologyspecimens) and/or metadata, and using machine learning or a rules basedsystem for determining a lifestyle recommendation.

The data ingestion tool 102 refers to a process and system forfacilitating a transfer of the digital pathology images to the varioustools, modules, components, and devices that are used for classifyingand processing the digital pathology images, according to an exemplaryembodiment.

The slide intake tool 103 refers to a process and system for scanningpathology images and converting them into a digital form, according toan exemplary embodiment. The slides may be scanned with slide scanner104, and the slide manager 105 may process the images on the slides intodigitized pathology images and store the digitized images in storage106.

The viewing application tool 108 refers to a process and system forproviding a user (e.g., a pathologist) with specimen property or imageproperty information pertaining to digital pathology image(s), accordingto an exemplary embodiment. The information may be provided throughvarious output interfaces (e.g., a screen, a monitor, a storage device,and/or a web browser, etc.).

The slide analysis tool 101, donor recipient inference tool 141, anddietary and lifestyle tool 144 and each of their components, maytransmit and/or receive digitized slide images and/or patientinformation to server systems 110, physician servers 121, hospitalservers 122, clinical trial servers 123, research lab servers 124,and/or laboratory information systems 125 over an electronic network120. Further, server systems 110 may include one or more storage devices109 for storing images and data received from at least one of the slideanalysis tool 101, the donor recipient inference tool 141, the dietaryand lifestyle tool 144, the data ingestion tool 102, the slide intaketool 103, the slide scanner 104, the slide manager 105, and viewingapplication tool 108. Server systems 110 may also include processingdevices for processing images and data stored in the storage devices.Server systems 110 may further include one or more machine learningtool(s) or capabilities, e.g., due to the processing devices.Alternatively or in addition, the present disclosure (or portions of thesystem and methods of the present disclosure) may be performed on alocal processing device (e.g., a laptop).

Any of the above devices, tools and modules may be located on a devicethat may be connected to an electronic network 120, such as the Internetor a cloud service provider, through one or more computers, servers,and/or handheld mobile devices.

FIG. 1C illustrates an exemplary block diagram of a slide analysis tool101, according to an exemplary embodiment of the present disclosure. Theslide analysis tool may include a training image platform 131 and/or aninference platform 135.

The training image platform 131, according to one embodiment, may createor receive training images that are used to train a machine learningsystem to effectively analyze and classify digital pathology images. Forexample, the training images may be received from any one or anycombination of the server systems 110, physician servers 121, hospitalservers 122, clinical trial servers 123, research lab servers 124,and/or laboratory information systems 125. Images used for training maycome from real sources (e.g., humans, animals, etc.) or may come fromsynthetic sources (e.g., graphics rendering engines, 3D models, etc.).Examples of digital pathology images may include (a) digitized slidesstained with a variety of stains, such as (but not limited to) H&E,Hematoxylin alone, IHC, molecular pathology, etc.; and/or (b) digitizedimage samples from a 3D imaging device, such as micro-CT.

The training image intake module 132 may create or receive a datasetcomprising one or more training images corresponding to either or bothof images of a human and/or animal tissue and images that aregraphically rendered. For example, the training images may be receivedfrom any one or any combination of the server systems 110, physicianservers 121, and/or laboratory information systems 125. This dataset maybe kept on a digital storage device. The training slide module 133 mayintake training data that includes images and corresponding information.For example, training slide module 133 training data may includereceiving one or more images (e.g., WSIs) of a human or animal. Thisdataset may be kept on a digital storage device. In some examples, thedataset may be comprised of a plurality of data subsets, where each datasubset corresponds to a training case from a plurality of training casesand includes one or more training images from the training case. Thetraining slide module 133 may include one or more computing devicescapable of, e.g., determining whether the training images have asufficient level-of-quality for training a machine learning model. Thetraining slide module 133 may further include one or more computingdevices capable of, e.g., identifying whether a set of individual cellsbelong to a cell of interest or a background of a digitized image.

The slide background module 134 may analyze images of tissues anddetermine a background within a digital pathology image. It is useful toidentify a background within a digital pathology slide to ensure tissuesegments are not overlooked.

According to one embodiment, the inference platform 135 may include anintake module 136, an inference module 137, and an output interface 138.The inference platform 135 may receive a plurality of electronicimages/additional information and apply one or more machine learningmodel to the received plurality of electronic images/information toextract relevant information and integrate spatial and orientationinformation for display on medical digital images. For example, theplurality of electronic images or additional information may be receivedfrom any one or any combination of the server systems 110, physicianservers 121, hospital servers 122, clinical trial servers 123, researchlab servers 124, and/or laboratory information systems 125. The intakemodule 136 may receive WSI's corresponding to one or morepatients/individuals. Further, the WSI's may correspond to an animal.The inference module 137 may apply one or more machine learning modelsto a group of WSI and any additional information in order to extractrelevant information and integrate spatial and orientation informationfor display on medical images. The inference module 137 may furtherincorporate the spatial characteristics of the salient tissue into theprediction.

The output interface 138 may be used to output information about theinputted images and additional information (e.g., to a screen, monitor,storage device, web browser, etc.). The output information may includeinformation related to ranking causes of death. Further, outputinterface 138 may output WSI's that indicate locations/salient regionsthat include evidence related to outputs from inference module 137.

Techniques discussed herein may use AI technology, machine learning,and/or image processing tools applied to patient and donor data, e.g.,digital images of histology slides, radiology, clinical reports, geneticinformation, microbial composition, life history, etc., to identifyoptimal donors for a patient in need of a transplant.

FIG. 2 illustrates an exemplary process 200 for to determining a donorrecipient prediction, according to techniques presented herein. Thesystems and methods disclosed herein may include data ingestion 202, adonor recipient inference module 204, and a dietary and lifestyleprediction module 206. The process described in FIG. 2 may be performedby the slide analysis tool 101, the donor recipient inference tool 141,and/or the dietary and lifestyle tool 144. In other examples, aspects ofthe system described in FIG. 2 may be performed in external systems andreceived by the tissue viewing platform 100.

In FIG. 2 , the system may first include data ingestion 202. Dataingestion 202 may be performed by the slide analysis tool 101. Dataingestion may include receiving one or more digital images (e.g., wholeslide image (WSI) of histopathological slide, magnetic resonance imaging(MRI), computed tomography (CT), positron emission tomography (PET),mammogram, ultrasound, X-rays, photographs of external anatomy, etc.)into a digital storage device 109 (e.g., hard drive, network drive,cloud storage, RAM, etc.). The digital medical images may include atleast digital medical images of transplant recipients (e.g., patientshaving previously received transplants from donors). The digital medicalimages may, in some examples, also include digital medical images oftransplant donors. The digital medical images may, in some examples,also include digital medical images of individuals that may need atransplant. The digital medical images may be digital medical images ofhuman tissues, organs, or other foreign matter associated with thetransplant, for example, and/or animal tissues, organs, or other foreignmatter (e.g., pig tissue, pig valves) that may be transplanted intohumans as part of the transplant procedure. Optionally, metadataassociated with the recipient and/or donor can also be ingested, e.g.,metadata such as body mass index (BMI), percentage of body fat, age,gender, ethnicity, microbial composition, genomics (e.g., longevityregulating pathway alterations), transcriptomics, metabolomics,ancillary test results, etc. For training the machine learning system,each image may be paired with information from the donors and/orrecipient, including whether the given transplant from the donor to therecipient was successful or not. Optionally, at least the digitalmedical images of the data ingested may be pre-processed. For example,one or more areas of tissue present in a digital medical image that mayhave been affected by the tissue biopsy or extraction procedure may beexcluded. As one illustrative example, during the tissue biopsy, thesurgeon performing the biopsy may introduce foreign microbes whencutting into the tissue. The area where the foreign microbes areintroduced may be removed from the digital image during pre-processingto prevent the presence of the foreign microbes from interfering withthe analysis of the image (e.g., by salient region detection moduleand/or the donor recipient inference tool 141, described below).

Next, data from the data ingestion 202 may be inserted into a salientregion detection module described in greater detail below in FIGS. 3Aand 3B. A salient region detection module, as further described below,may be used to identify the salient regions to be analyzed for eachdigital image. This may be done manually by a human or automaticallyusing AI. An entire image or specific image regions can be consideredsalient. Salient region determination techniques are discussed in U.S.application Ser. No. 17/313,617, which is incorporated by referenceherein in its entirety.

Next, the digital medical images from the data ingestion 202, which mayor not have had a salient region identified, may be fed to a donorrecipient inference module 204 (e.g., the donor recipient inference tool141). The donor recipient inference module 204 that may implement atrained machine learning model to predict one or more optimal donors fora patient in need of a transplant. The model may incorporate spatialinformation from disparate regions in a digital medical image of thepatient (e.g. may produce an embedding representing attributes of thepatient in an embedding space) to facilitate the prediction. Theprediction may output to an electronic storage device.

Next, the digital medical images from the data ingestion 202, which mayor not have had a salient region identified and may or may not have hadthe donor recipient inference module 204 applied, are fed to a dietaryand lifestyle prediction module 206 (e.g., the dietary and lifestyletool 144). The dietary and lifestyle prediction module 206 may implementa trained machine learning model to predict one or more predictlifestyle recommendations. The model may incorporate spatial informationfrom disparate regions in a digital medical image of the patient (e.g.may produce an embedding representing attributes of the patient in anembedding space) as well as received metadata to facilitate theprediction. The prediction is output to an electronic storage device.

The optional salient region detection module, the donor recipientinference module 204, and the dietary and lifestyle prediction module206 are described in turn below.

One aspect of the systems and methods disclosed herein includes theautomatic identification of one or more salient regions to be analyzedfor a digital image using AI. This may be performed by a salient regiondetection module. An entire image or specific image regions may beconsidered salient.

A continuous score of interest may be specific to certain structureswithin the digital image, and it can be important to identify relevantregions so that they can be included while excluding irrelevant ones.For example, with MRI, PET, or CT data localizing a specific organ ofinterest could be needed. Salient region identification can enable thedownstream machine learning system to learn how to detect morphologiesfrom less annotated data and to make more accurate predictions.

The salient region detection module can output a salient region that wasspecified by a human annotator using an image segmentation mask, abounding box, line segment, point annotation, freeform shape, or apolygon, or any combination of the aforementioned. Alternatively, thismodule can be created using machine learning to identify the appropriatelocations.

As described in more detail below with respect to the steps performed totrain one or more machine learning systems to identify one or moresalient regions of a digital image, there are two general approaches tousing machine learning to create a salient region detector. The firstapproach includes strongly supervised methods that identify preciselywhere the morphology of interest could be found. The second approachincludes weakly supervised methods that do not provide a preciselocation.

For strongly supervised training, the system needs the image and thelocation of the salient regions that could potentially express thebiomarker as input. For 2D images, e.g., whole slide images (WSI) inpathology, these locations could be specified with pixel-level labeling,bounding box-based labeling, polygon-based labeling, or using acorresponding image where the saliency has been identified (e.g., usingimmunohistochemical (IHC) staining). For 3D images, e.g., CT and MRIscans, the locations could be specified with voxel-level labeling, usinga cuboid, etc. or use a parameterized representation allowing forsubvoxel-level labeling, such as parameterized curves or surfaces, ordeformed template. For weakly supervised training, the system requiresthe image or images and the presence/absence of the salient regions, butthe exact location of the salient location does not need to bespecified.

The training of the salient region detection module discussed in FIG. 2may be described in greater detail below. Examples of training thesalient region detection module may include method 300 of FIG. 3A.Examples of using the salient region detection module may include method350 of FIG. 3B.

FIG. 3A is a flowchart illustrating an example method 300 of training analgorithm for region detection, according to an exemplary embodiment ofthe present disclosure. The method 300 of FIG. 3A depicts steps that maybe performed by, for example, the slide analysis tool 101 as describedabove in FIG. 1C. Alternatively, the method may be performed by anexternal system. According to one example aspect, for training the oneor more machine learning systems to identify one or more salient regionsof a digital image, the following method 300 may be performed:

At step 302, the system may receive one or more digital images of amedical specimen (e.g., histopathological slide images, CT, MRI, PET,mammogram, ultrasound, X-rays, photographs of external anatomy, etc.)into a digital storage device (e.g., hard drive, network drive, cloudstorage, RAM, etc.) and an indication of the presence or absence of thesalient region (e.g., a particular organ, tissue, region of tissue,etc.) within the image.

At step 304, the system may, break each digital image into sub-regionsthat will then have their saliency determined. Regions can be specifiedin a variety of methods, including creating tiles of the image,segmentations based on edge/contrast, segmentations via colordifferences, segmentations based on energy minimization, superviseddetermination by the machine learning model, EdgeBoxes, etc.

At step 306 a machine learning system may be trained that takes as inputa digital image and predicts whether the salient region is present ornot. Training the salient region detection module may also includetraining a machine learning system to receive, as an input, a digitalimage and to predict whether the salient region is present or not. Manymethods may be used to learn which regions are salient, including butnot limited to weak supervision, bounding box or polygon-basedsupervision, or pixel-level or voxel-level labeling.

Weak supervision may involve training a machine learning model (e.g.,multi-layer perceptron (MLP), convolutional neural network (CNN),transformers, graph neural network, support vector machine (SVM), randomforest, etc.) using multiple instance learning (MIL). The MIL may useweak labeling of the digital image or a collection of images. The labelmay correspond to the presence or absence of a salient region.

Bounding box or polygon-based supervision may involve training a machinelearning model (e.g., R-CNN, Faster R-CNN, Selective Search, etc.) usingbounding boxes or polygons. The bounding boxes or polygons may specifysub-regions of the digital image that are salient for detection of thepresence or absence of a biomarker.

Pixel-level or voxel-level labeling (e.g., semantic or instancesegmentation) may involve training a machine learning model (e.g., MaskR-CNN, U-Net, fully convolutional neural network, transformers, etc.)where individual pixels and/or voxels are identified as being salientfor the detection of continuous score(s) of interest. Labels couldinclude in situ tumor, invasive tumor, tumor stroma, fat, etc.Pixel-level/voxel-level labeling may be from a human annotator or may befrom registered images that indicate saliency.

According to another example aspect, to implement the one or moretrained machine learning systems for identifying one or more salientregions in a digital image, the following steps may be performed:

FIG. 3B is a flowchart illustrating methods 350 for how to provide imageregion detection, according to one or more exemplary embodiments herein.FIG. 3B may illustrate a method that utilizes the neural network thatwas trained in FIG. 3A. The exemplary method 350 (e.g., steps 352-356)of FIG. 3B depicts steps that may be performed by, for example, by theslide analysis tool 101. These steps may be performed automatically orin response to a request from a user (e.g., physician, pathologist,etc.). Alternatively, the method described in flowchart 350 may beperformed by any computer process system capable of receiving imageinputs such as device 900 and capable of including or importing theneural network described in FIG. 3A.

At step 352, a system may receive one or more digital medical images maybe received of a medical specimen into a digital storage device (e.g.,hard drive, network drive, cloud storage, RAM, etc.). Using the salientregion detection module may optionally include breaking or dividing eachdigital image into sub-regions and determining a saliency (e.g.,sub-regions of tissue which has morphology of interest) of eachsub-region using the same approach from training step 304.

At step 354, the trained machine learning system from FIG. 3A may beapplied to the inputted images to predict which regions of the image aresalient and could potentially exhibit the continuous score(s) ofinterest.

At step 356, if salient regions are found at step 354, the system mayidentify the salient region locations and flag them. If salient regionsare present, detection of the region can be done using a variety ofmethods, including but not restricted to: running the machine learningmodel on image sub-regions to generate the prediction for eachsub-region; or using machine learning visualization tools to create adetailed heatmap, etc. Example techniques are described in U.S.application Ser. No. 17/016,048, filed Sep. 9, 2020, and Ser. No.17/313,617, filed May 6, 2021, which are incorporated herein byreference in their entireties. The detailed heatmap may be created byusing class activation maps, GradCAM, etc. Machine learningvisualization tools may then be used to extract relevant regions and/orlocation information.

The salient regions may be any tissue regions. For example, the salientregion could correspond to lamina propria (Mucous membrane), orsubmucosa.

The outputted salient regions from step 356 may then be fed into thedonor recipient inference module 204. The training of the donorrecipient inference module 204 may be described in greater detail below.Examples of training the donor recipient inference module 204 mayinclude method 400 of FIG. 4A. Examples of using donor recipientinference module 204 may include method 450 of FIG. 4B.

Another aspect of donor recipient inference module 204 disclosed hereinincludes using AI technology, machine learning, and/or image processingtechniques to identify one or more optimal donors for a patient in needof a transplant.

FIG. 4A is a flowchart illustrating an example method 400 of training adonor recipient inference module according to an exemplary embodiment ofthe present disclosure. The method 400 of FIG. 4A depicts steps that maybe performed by, for example, the donor recipient inference tool 141 asdescribed above. Alternatively, the method 400 may be performed by anexternal system.

According to one example aspect, for training a machine learning modelto predict optimal transplant donors for a given transplant type, method400 may be performed.

At step 402, the system may receive training data into a digital storagedevice 109 (e.g., hard drive, network drive, cloud storage, RAM, etc.).The training data may include a plurality of donor-recipient pairprofiles associated with previously performed transplants of a giventype. Each donor-recipient pair profile may include metadata associatedwith a donor of a transplant and a recipient of the transplant (e.g.,oxygen at rest, pulmonary artery pressure, age, body mass index (BMI),total bilirubin), as well as an outcome of the transplant (e.g.,successful or not successful, rejected or not rejected, graft survivalor graft failure, survival time following transplantation, and othersimilar measures, including total lung capacity (TLC), peak oxygenconsumption, and oxygen saturation). The particular types of metadataand/or outcome metrics may be based on a type of the transplant. In someexamples, depending on a type of the transplant, a delivery type usedfor each donor-recipient pair profile may be included (e.g., for FMT,pill versus liquid). Information from the donor and recipient may eitherbe an electronically documented text paragraph, structured data, imagingdata or numbers that is stored in and received from a digital storagedevice (e.g., hard drive, network drive, cloud storage, RAM, etc.).Example types of metadata associated with the donor and/or the recipientof the transplant may include digital images (e.g., whole slide imagesof histopathological slides and/or radiological images), clinical data,genetic information, microbial composition, life history, and/or othersimilar data. For example, the digital images may be of intestinaltissue. The tissue may be of areas of a colon cecum, ascending,transverse, descending, sigmoid, or a rectum. In some examples, aportion of the metadata may be ingested to stratify and split the systemfor machine learning. The metadata may further include digital medicalimages of fecal matter. The plurality of donor-recipient pair profilesmay be stored within a database for subsequent lookup and/or reference.At least a portion of the data from the plurality of donor-recipientpair profiles may be used as training data for a machine learning model.

Next, the salient region detection module described in FIGS. 3A and 3Bmay be applied to identify the saliency of each region within thedigital medical images (e.g., of histology images) and excludenon-salient image regions from subsequent processing.

At step 404, the system may generate and train the machine learningsystem using the training data. The trained machine learning system maybe located within the donor recipient inference tool 141. The trainedsystem may have two parts, an embedding tool 142 and a donorrecommendation tool 143, both located within the donor recipientinference tool 141. The machine learning model may be built and trainedusing the training data from step 402. For example, the machine learningmodel may output, for at least the recipients of the donor-recipientpair profiles, recipient embeddings based on the respective digitalmedical images and/or other metadata of the recipients in an embeddingspace (e.g., a vector). The recipient embeddings may be determined bythe embedding tool 142. The recipient embeddings may indicate aplurality of attributes representing the respective recipients, suchthat recipients that share more similar attributes may be of a closerdistance to one another in the embedding space. By representing at leasteach of the recipients as vectors in the embedding space throughproduction of the embeddings, a donor recommendation system may be builtfor identifying optimal transplant donor-recipient matches. The donorrecommendation system may be performed by the donor recommendation tool143. The donor recommendation tool 143 may receive as input embeddingsdetermined by the embedding tool 142. In some examples, either theembedding tool 142 or the donor recommendation tool 143 may be locatedon an external server and accessed through network 120. Further,embedding tool 142 and donor recommendation tool 143 may be two separatemachine learning systems that are trained separately.

For example, for a patient in need of a transplant, the trained machinelearning model may produce a patient embedding in the embedding spaceand identify similar recipients to the patient having recipientembeddings close in distance from the patient embedding in the embeddingspace. If those identified similar recipients had a successfultransplant from the donors of the respective donor-recipient pairprofiles, the system may infer that those donors may also provide anoptimal match to the patient in need of the transplant (e.g., result ina successful transplant outcome for the patient). In some examples, themachine learning model(s) are trained using a semi-supervised, multipleinstance learning approach. In other examples, other machine learningapproaches (e.g., supervised, unsupervised, semi-supervised) may beutilized. Additionally or alternatively, when the metadata of the donorsof the donor-recipient pair profiles are included as training data, themachine learning model may output donor embeddings for the donors basedon the respective digital images and/or other metadata in the embeddingspace. The donor embeddings may similarly indicate a plurality ofattributes representing the respective donors. In such examples, for apatient in need of a transplant, the trained machine learning model mayproduce a patient embedding in the embedding space and identify donorssimilar to the patient (e.g., donors having donor embeddings closest indistance to the patient embedding in the embedding space). Based on thesimilarity between the patient and the donors, the system may infer thatthose donors may provide an optimal match to the patient in need of thetransplant (e.g., result in a successful transplant outcome for thepatient.

At step 406, the trained machine learning system may be saved in digitalstorage (e.g., digital storage 109) for subsequent use.

FIG. 4B is a flowchart illustrating an exemplary method of performing adonor recipient inference according to an exemplary embodiment of thepresent disclosure. The exemplary method 450 (e.g., steps 452-460) ofFIG. 4B depicts steps that may be performed by, for example, by thedonor recipient inference tool 141. These steps may be performedautomatically or in response to a request from a user (e.g., physician,pathologist, etc.). These steps may describe an exemplary method of howto use the trained system described in FIG. 4A. Alternatively, themethod described in flowchart 450 may be performed by any computerprocess system capable of receiving image inputs such as device 900 andcapable of including or importing the neural network described in FIG.4A.

According to one example aspect, to implement the trained machinelearning model to predict optimal transplant donors for a giventransplant type, the method 450 may be performed.

At step 452, the system may determine a trained machine learning system.For example, the trained learning system may be the machine learningsystem described in FIG. 4A. The trained system may perform a donorrecipient inference. In one example, the trained system may beimplemented by the donor recipient inference tool 141. In anotherexample, the trained machine learning system may be imported throughnetwork 120.

At step 454, the system may receive at least a digital medical image(e.g., a whole slide image and/or radiology image) of a patient in needof a transplant of the given type from a donor. The digital medicalimage may include at least a tissue, or other anatomical structureassociated with the transplant. For example, if the transplant is anFMT, colon tissue may be biopsied, and a histopathological slide may beprepared and imaged. As another example, if the transplant is a livertransplant, liver tissue may be biopsied, and a histopathological slidemay be prepared and imaged. In another example, the trained system mayalso receive a digital medical image of fecal matter. Optionally,additional metadata associated with the patient may be received, such asclinical data, genetic information, microbial composition, life history(including current lifestyle), and/or other similar data.

Next, the trained system may apply the salient region detection moduledescribed in FIGS. 3A and 3B to identify the saliency of each regionwithin the digital medical image received and exclude non-salient imageregions from subsequent processing.

At step 456, the digital medical image and/or the metadata from step 454may be received by the trained machine learning model received at step452. The trained machine learning model may produce a patient embeddingin the embedding space based on the digital medical image. If theadditional metadata is optionally received, the trained machine learningmodel may also produce the patient embedding based on the additionalmetadata. Using the patient embedding, the trained machine learningmodel may identify a subset of recipients (e.g., from the recipients ofthe donor-recipient profile pairs having previously received atransplant of the given type) that are similar to the patient. Forexample, the subset of recipients identified may include one or morerecipients having recipient embedding(s) within a threshold distance ofthe patient embedding in the embedding space. In some examples, thesubset of recipients identified as being similar to the patient may befurther refined to exclude those recipients whose transplant was notsuccessful. The one or more donors of the transplants for the one ormore recipients remaining in the further refined subset may beidentified as optimal donor(s) (e.g., the donors identified byreferencing the database storing the donor-recipient pair profiles) andoutput as a recommendation. This recommendation may be referred to as adonor recipient inference. In examples where digital medical images ofthe donors are received and used to produce donor embeddings in theembedding space, in addition to or rather than identifying similarrecipients, similar donors may be identified. For example, one or moredonors having donor embedding(s) within a threshold distance of thepatient embedding in the embedding space may be identified.

At step 458, the system may receive the recommendation of optimaldonor(s) as output of the trained machine learning model. In someexamples, when there is more than one optimal donor identified andrecommended, the optimal donors may be ranked. In some examples, thedonors may be ranked based on a mathematical distance between therespective recipient embeddings and the patient embedding in theembedding space. For example, a donor of the most similar recipient isranked highest. Additionally and/or alternatively, the donors may beranked on additional metadata associated with the donor-recipient pairprofiles (e.g., stored in the database). The additional metadata mayinclude qualitative data related to an outcome of the recipient of thedonor-recipient pair. For example, a donor of a donor-recipient pairwhose recipient had a quicker recovery, a longest survival rate, and/orrequired the least amount of immunosuppressant to prevent rejection, maybe ranked highest. The additional metadata may also include qualitativedata related to a health of the donor of the donor-recipient pair. Forexample, a donor having a better health provided based on exerciseperformance, overall fitness, lung capacity, age, and/or other similarhealth factors may be ranked highest.

At step 460, the system may save the recommendation to a digital storage109 (e.g., to a medical record associated with the patient). In someexamples, the recommendation may be transmitted to an electronic healthcare record system to be included (e.g., stored within) a medical recordassociated with the patient. This may include transmitting theprediction by electronic network 120 to either the hospital servers 122,the research lab server 124, laboratory information systems 125, thephysician servers 121, or clinical trial servers 123. In furtherexamples, the prediction may be provided as input to other systems suchas transplant databases.

In some examples (e.g., dependent on a type of the transplant), inaddition to recommending an optimal donor, the system may also betrained and implemented to recommend a delivery type for the transplant.For example, for an FMT, the fecal microbiota may be transplanted in apill form orally consumed by the patient or in a liquid form insertedrectally into the gastrointestinal tract. In such examples, the trainingdata used to train the machine learning model may include dataassociated with the delivery type used for each donor-recipient pair.

The dietary and lifestyle prediction module 206, described in FIGS. 5Aand 5B below, may be applied to a patient in response to determining arecommendation from the donor recipient inference module 204. In anotherexample, the dietary and lifestyle prediction module 206 may be appliedas a separate system. Other techniques described herein used inconjunction or separately with the above-described identification ofoptimal transplant donor-recipient matches may include application of AItechnology, machine learning, and/or image processing tools to providelifestyle recommendations (e.g., diet and exercise recommendations) toimprove a patient's quality of life and/or extend their lifespan.

FIG. 5A is a flowchart illustrating an example method 500 of training adietary and lifestyle prediction module 206 according to an exemplaryembodiment of the present disclosure. The method 500 of FIG. 5A depictssteps that may be performed by, for example, the dietary and lifestyletool 144 as described above. Alternatively, the method 500 may beperformed by an external system.

According to one example aspect for training a machine learning model topredict lifestyle recommendations, method 500 may be performed.

At step 502, the system may receive training data into a digital storagedevice 109 (e.g., hard drive, network drive, cloud storage, RAM, etc.).The training data may include metadata for a plurality of patients whohad previously received/implemented lifestyle recommendations. Themetadata may include BMI, percentage of body fat, age, gender,ethnicity, pathology WSI, radiology images, microbial composition,genomics (e.g., longevity regulating pathway alterations),transcriptomics, and/or metabolomics. The metadata may also include aninitial set of lifestyle attributes associated with the patient, such asa diet and/or exercise of the patient prior to the patientreceiving/implementing the lifestyle recommendations. In some examples,the patients include recipients of a particular type of transplant(e.g., FMT recipients, heart transplant recipients, kidney transplantrecipients, etc.) and/or recipients at a high risk for a particular typeof disease (e.g., heart disease, cancer, etc.). In some examples, thetraining data may also include labels associated with specific lifestyleattributes associated with diet, exercise, etc. that improved (and/oralternatively did not improve) a quality of life of the patient.

At step 504, the system may build and train a machine learning model topredict a lifestyle recommendation using the training data from step502. As part of the training process, for the metadata of a givenpatient, the machine learning model may learn to characterize themetadata into embeddings, evaluate the embeddings for a signal or acluster of signals, and predict a lifestyle recommendation based on thesignal or cluster of signals. In some examples, the characterization ofthe patient's metadata into embeddings may include at leastcharacterizing morphology of available tissue such as fatty tissue,muscle tissue, stromal tissue, necrotic tissue, cancerous tissue,inflamed tissue etc. (e.g., from the pathology WSI and/or radiologyimages of the training data). In some examples, the machine learningmodel is trained using a semi-supervised, multiple instance learningapproach. In other examples, other machine learning approaches (e.g.,supervised, unsupervised, semi-supervised) may be utilized. In someexamples, the system may include a first tool for determining embeddingsand a second tool for analyzing the embeddings and outputting arecommendation.

At step 506, the system may save the trained machine learning model toelectronic storage (e.g., electronic storage 109).

According to one example aspect, to implement a trained learning modelfor predicting a lifestyle recommendation for a target patient, method550 may be performed.

FIG. 5B is a flowchart illustrating an exemplary method of performing adietary and lifestyle prediction according to an exemplary embodiment ofthe present disclosure. The exemplary method 550 (e.g., steps 552-560)of FIG. 5B depicts steps that may be performed by, for example, by thedietary and lifestyle tool 144. These steps may be performedautomatically or in response to a request from a user (e.g., physician,pathologist, etc.). These steps may describe an exemplary method of howto use the trained system described in FIG. 5A. Alternatively, themethod described in flowchart 550 may be performed by any computerprocess system capable of receiving image inputs such as device 900 andcapable of including or importing the neural network described in FIG.5A. The machine learning system of method 550 may be implemented todetermine dietary and/or lifestyle suggestions for a target patient. Thetarget patient may, for example, be in need of a transplant.

At step 552, the system may receive the trained machine learning modelfrom FIG. 5A.

At step 554, the system may receive metadata for a target patient, suchas BMI, percentage of body fat, age, gender, ethnicity, pathology WSI,radiology images, microbial composition, genomics (e.g., longevityregulating pathway alterations), transcriptomics, and/or metabolomics.The metadata may also include a current state associated with the targetpatient's lifestyle (e.g., current diet, current exercise, etc.).

At step 556, the system may provide the metadata as input to the trainedmachine learning model received at step 552. The trained machinelearning model may characterize morphology of available tissue such asfatty tissue, muscle tissue, stromal tissue, necrotic tissue, canceroustissue, inflamed tissue etc. (e.g., from the pathology WSI and/orradiology images) into embeddings. Additionally, the remaining metadatamay be characterized into embeddings. These embeddings may be evaluatedby the trained machine learning model for a signal or a cluster ofsignals. For example, this may be performed by techniques such ask-means clustering. Based on the signal or cluster of signalsidentified, the trained machine learning model may provide, as output, alifestyle recommendation associated with one or more diet and/orexercise regimens that could lead to improvements in the patient'squality of life (e.g., increase lifespan).

At step 558, the system may receive the lifestyle recommendation outputby the trained machine learning model. Life system recommendations mayinclude: particular exercise regimes, a dietary output (e.g., foods toavoid and include in one's diet, food portions to eat, etc.), sleepschedules, suggested physical fitness objectives/goals, etc. Therecommendation output may, for example, determine that the muscle tissueon the digital medical images looks insufficient and then output arecommendation to increase/supplement protein intake (e.g., a dietaryoutput).

At step 560, the system may save the lifestyle recommendation to digitalstorage 109 (e.g., store in an electronic medical record of thepatient).

In one example, when a specific fatty tissue morphology in a specificorgan of a patient is identified, the lifestyle recommendation mayinclude a specific diet and/or exercise regimen to reduce the amount offatty tissue. Implementation of this recommendation by the patient maylead to weight loss and/or a reduction in the progression or developmentof future diseases such as cancer. In another example, for an FMTrecipient, a lifestyle recommendation may be provided to specificallymaintain the health of the patients' microbiome post-transplant.

Similar techniques may also be applied to analyze the microbiomes ofdonors over time in terms of diet, lifestyle, exercise, etc. todetermine which lifestyle types maintain optimal bioflora. In somescenarios, these optimal lifestyle types may be used to further rankdonors determined to be optimal for a patient in need of a transplant bythe above-described systems and methods for identifying optimaltransplant donor-recipient matches. For example, a donor having alifestyle type determined to maintain optimal bioflora may be selectedover another donor having a less optimal lifestyle type.

Other techniques described herein used in conjunction or separately withthe above-described identification of optimal transplant donor-recipientmatches and/or providing of lifestyle recommendations may includeautomatically notifying donor collection entities, e.g., hospitals, of aneed for donors with a given profile. This may be performed by, forexample, the tissue viewing platform 100. As part of the automaticnotification system, a patient at a hospital who meets the givenprofile, among other requirements for transplant, could be flagged toprompt a medical professional to discuss with the patient if they wouldbe willing to be a donor for the relevant application, e.g., livertransplant, fecal transplant, kidney transplant, etc. Additionally oralternatively, candidate donors for critical organ removal (e.g., braindead or very recently deceased individuals) meeting the given profilemay be matched.

For example, when an optimal donor is identified for a patient in needof a transplant using the above-described systems and methods foridentifying optimal transplant donor-recipient matches, an additionalevaluation may be performed to determine whether to trigger automaticnotification. As one example, if the output of the system foridentifying optimal transplant donor-recipient matches is a list ofoptimal donors, but each of those donors have already donated and/or arenot applicable for a second donation, the automatic notification may betriggered. The notification may include metadata of the patient in need,an indication of the donor profile needed that matches the recipient,and/or a request for the donor collection entities to start collectingdonors that have a similar profile.

In other examples, a threshold for triggering the automatic notificationmay be based on a list of donors having a given profile being less thanN. For example, the system may periodically check a database listingdonors and their associated donor profiles for a given type oftransplant. Based on the periodic checks, if the number of donors havinga given profile are below a threshold number or, in the case of FMTs, ifit is determined that there are limited (or no) samples remaining in astool bank for the optimal donor and/or donors having a similar profileto the optimal donor, the automatic notification may be triggered.

In further examples, patterns or trends in transplant needs may belearned overtime, and the automatic notification may be triggeredproactively to ensure that there are sufficient donors and/or donorsamples to fulfill the needs. Example patterns or trends learned mayinclude an increase at a healthcare facility for transplant requestsfrom recipients or less transplant donors volunteering, which may promptthe healthcare facility to proactively increase donor outreach andawareness programs. Other example patterns or trends learned may includepopulation dietary trends, where a trend of popularity of a dietincorporating a particular food product that may lead to an increase intransplant need may be measured in order to predict future number ofdonors to fulfill transplant needs. Further example patterns or trendslearned may be associated with publications and/or new transplant methodtrials or approvals. A separate machine learning model may be trained tolearn the patterns or trends over time to predict future demand. Thismay be performed by, for example, slide analysis tool 101.Alternatively, this may be performed by an external system capable ofreceiving image inputs such as device 900. For example, inputs to themachine learning model may include data collected from a plurality ofresources such as: a number of recipients and/or donors or adonor-recipient ratio from healthcare facilities; sales data on foodsand beverages from food and beverage retailers and/or social mediapopularity data associated with foods and beverages; population-basedfeedback and/or publication and regulatory announcement from entitiessuch as the National Wastewater Surveillance System (e.g., measurementsof microbes, metabolites or viruses in waste water) and the Food andDrug Administration; and/or drug sales data (e.g., indicating acorrelation between a particular medication use and a transplantincrease).

FIG. 6 illustrates an exemplary process 600 for determining a FMT donorrecipient, according to techniques presented herein. One example usecase or application of one or more of the above-described systemsincludes a scenario where a patient (e.g., patient 1) needs an FMT totreat a gastrointestinal disease, such as Clostridium difficile colitis.The patient may undergo a colon polyp biopsy and/or colon resection. Awhole slide image (WSI) 602 of the colon tissue biopsied and/or resectedmay be created. At least the WSI 602 may be provided as input to theabove-described system for identifying an optimal transplantdonor-recipient match. For example, one or more WSI 602 (patient 1 WSI)may be provided as input to a trained machine learning model (e.g., thedonor recipient inference tool 141) and a colon biopsy embedding 604 maybe determined. The colon biopsy embedding 604 determined from the WSI602 may be based on an analysis of the morphology and structure of thetissue to indicate attributes thereof (and optionally other patient 1metadata provided as input to the trained machine learning model). Thetrained machine learning model may be implemented by, for example, thedonor recipient inference tool 141. The trained machine learning modelmay utilize a similarity function to identify a subset of recipients(e.g., from recipients of donor-recipient pair profiles stored in adatabase and used to train the model) that have previously undergone anFMT from donors and are similar to patient 1. For example, colon biopsyembeddings may have been learned for the recipients in the embeddingspace based on colon biopsy WSIs for the recipients as part of thetraining process, and a similarity function may be performed to indicatehow similar a given recipient embedding is to the embedding for patient1 based on a distance between the embeddings in the embedding space.Patient 2 may be an identified recipient donor. Patient 2 may havepreviously performed a colon polyp biopsy and/or colon resection anddetermined a WSI 612. As illustrated, a colon biopsy embedding 614(determined by the trained machine learning model) for patient 2 (e.g.,based on the patient 2's WSI 612) may be determined to be similar 606(e.g., within a threshold distance in the embedding space) to the colonbiopsy embedding for patient 1. This may be performed by the trainedmachine learning model. In some examples, one or more other recipientsmay be identified in addition to patient 2 within the subset as beingsimilar to patient 1. From the subset of recipients identified, thoserecipients who had a successful transplant outcome may be furtheridentified (e.g., by referencing the database storing thedonor-recipient pair profiles), and donors for those patients may beprovided as recommended optimal donor(s) for the target recipient. Thismay include for example donor 1 608. For example, at least patient 2identified being as similar patient 1 may have had a successful outcome,and thus the donor of patient 1's FMT is identified from the informationwithin the database and is provided in the recommendation. Informationregarding patient 2's donor 1 608 and the received fecal mattertransplant 610 may be recorded and saved.

When more than one optimal donor is identified, the donors may be rankedbased a mathematical distance between the recipient embedding of therecipient from the donor-recipient pair (e.g., patient 2) and thepatient embedding of the (e.g., patient 1) in the embedding space. Forexample, a donor of the most similar patient is ranked highest (e.g.,this may be referred to as donor 1 608). Additionally and/oralternatively, the donors may be ranked on additional metadataassociated with donor-recipient pair (e.g., stored in the database). Theadditional metadata may include qualitative data related to an outcomeof the patient of the donor-recipient pair. For example, a donor of adonor-recipient pair whose recipient had a quicker recovery, had alongest survival rate, and/or patient required the least amount ofimmunosuppressant to prevent rejection, among other similar examples,may be ranked highest.

Further, for FMTs, donor collection entities, e.g., hospitals, mayinclude a stool bank that includes a plurality of stool samples fromeach donor. In some examples, the stool of the donors may undergomicrobiome sequencing to identify the microbial composition of thedonor′ gastrointestinal tracts. Once an optimal donor, e.g., donor 1, isidentified/recommended, a sample of the donor's stool (e.g., FMT 610)may be retrieved from the bank for use in the FMT procedure. Inscenarios where no stool samples for the optimal donor remain, asubstitute donor may be determined by identifying a stool sample havingsimilar microbiome sequencing to the stool sample of the optimal donor.In some examples, these types of scenarios may also trigger an automaticnotification to the donor collection entities to prompt collection ofmore stool samples, if available, from the optimal donor (e.g., donor 1608) and/or from other donors having a similar profile to the optimaldonor.

FIG. 7 illustrates an exemplary process 700 for to determining adietary, lifestyle and lifespan recommendations, and/or a delivery typerecommendation, according to techniques presented herein. FIG. 7 mayinclude receiving one or more trained machine learning systems 712(e.g., the donor recipient inference tool 141 and the dietary andlifestyle tool 144) being determined and/or received. The trainedmachine learning systems 712 may collectively be referred to as a WSIand multi-omic consumption system. The one or more trained machinelearning systems 712 may then, for example through network 120, receiveinformation related to a patient. The patient may be in need of a donortransplant. The one or more trained machine learning systems 712 mayfirst receive a colon polyp WSI 704, and/or a colon resection WSI 706for a patient. The one or more trained machine learning systems mayreceive additional biopsy WSIs.

In addition to the colon polyp WSI 704 and/or resected colon WSIs 706 ofthe patient (patient 1) discussed above, the one or more trained machinelearning systems 712 can consume additional metadata 702 of the patientsuch as body fat composition, BMI, etc. Furthermore, the one or moretrained machine learning systems 712 may also take in current microbialcomposition (e.g., 16S) 708 and metabolomics 710. The one or moretrained machine learning systems 712 may then determine embeddings forthe received information. In one example, the system may determine afirst set of embeddings related to a donor profile and a second set ofembeddings related to the patient's diet and lifestyle.

Based on the first set of embeddings, the one or more trained machinelearning systems 712 may determine a donor profile recommendation 716.Based on the second set of embeddings, the one or more trained machinelearning systems 712 may determine a dietary, lifestyle, and lifespanrecommendations 714.

Resultantly, in addition to predicting an optimal donor for the patient(e.g., a donor profile recommendation 716) and providing dietarylifestyle and lifespan recommendations 714, the one or more trainedmachine learning systems 712 may predict and recommend beneficialmicrobial profiles 718, and/or a delivery type recommendation 720 (pillvs. liquid) in relation to the microbial profile recommendation 720.

FIG. 8 illustrates an exemplary flowchart 800 for processing images todetermine a donor recipient prediction, according to techniquespresented herein.

At step 802, a digital medical image of the patient may be received,wherein the patient is in need of a transplant.

At step 804, a trained machine learning system may be determined.

At step 806, the digital medical image may be provided into the trainedmachine learning system, the trained machine learning system determininga patient embedding.

At step 808, using the patient embedding, a subset of donor recipientsmay be determined.

At step 810, based on the subset of donor recipients a recommendation ofoptimal donors may be determined.

FIG. 9 depicts an example of a computing device that may executetechniques presented herein, according to one or more embodiments.

As shown in FIG. 9 , device 900 may include a central processing unit(CPU) 920. CPU 920 may be any type of processor device including, forexample, any type of special purpose or a general-purpose microprocessordevice. As will be appreciated by persons skilled in the relevant art,CPU 920 also may be a single processor in a multi-core/multiprocessorsystem, such system operating alone, or in a cluster of computingdevices operating in a cluster or server farm. CPU 920 may be connectedto a data communication infrastructure 910, for example a bus, messagequeue, network, or multi-core message-passing scheme.

Device 900 may also include a main memory 940, for example, randomaccess memory (RAM), and may include a secondary memory 930. Secondarymemory 930, for example a read-only memory (ROM), may be, for example, ahard disk drive or a removable storage drive. Such a removable storagedrive may comprise, for example, a floppy disk drive, a magnetic tapedrive, an optical disk drive, a flash memory, or the like. The removablestorage drive in this example reads from and/or writes to a removablestorage unit in a well-known manner. The removable storage may comprisea floppy disk, magnetic tape, optical disk, etc., which is read by andwritten to by the removable storage drive. As will be appreciated bypersons skilled in the relevant art, such a removable storage unitgenerally includes a computer usable storage medium having storedtherein computer software and/or data.

In alternative implementations, secondary memory 930 may include similarmeans for allowing computer programs or other instructions to be loadedinto device 900. Examples of such means may include a program cartridgeand cartridge interface (such as that found in video game devices), aremovable memory chip (such as an EPROM or PROM) and associated socket,and other removable storage units and interfaces, which allow softwareand data to be transferred from a removable storage unit to device 900.

Device 900 also may include a communications interface (“COM”) 960.Communications interface 960 allows software and data to be transferredbetween device 900 and external devices. Communications interface 960may include a modem, a network interface (such as an Ethernet card), acommunications port, a PCMCIA slot and card, or the like. Software anddata transferred via communications interface 960 may be in the form ofsignals, which may be electronic, electromagnetic, optical or othersignals capable of being received by communications interface 960. Thesesignals may be provided to communications interface 960 via acommunications path of device 900, which may be implemented using, forexample, wire or cable, fiber optics, a phone line, a cellular phonelink, an RF link or other communications channels.

The hardware elements, operating systems, and programming languages ofsuch equipment are conventional in nature, and it is presumed that thoseskilled in the art are adequately familiar therewith. Device 900 mayalso include input and output ports 950 to connect with input and outputdevices such as keyboards, mice, touchscreens, monitors, displays, etc.Of course, the various server functions may be implemented in adistributed fashion on a number of similar platforms, to distribute theprocessing load. Alternatively, the servers may be implemented byappropriate programming of one computer hardware platform.

While the above-discussed use case describes the application of thesystem for an FMT, techniques presented herein may be applied in avariety of different transplant applications.

Throughout this disclosure, references to components or modulesgenerally refer to items that logically may be grouped together toperform a function or group of related functions. Like referencenumerals are generally intended to refer to the same or similarcomponents. Components and/or modules may be implemented in software,hardware, or a combination of software and/or hardware.

The tools, modules, and/or functions described above may be performed byone or more processors. “Storage” type media may include any or all ofthe tangible memory of the computers, processors or the like, orassociated modules thereof, such as various semiconductor memories, tapedrives, disk drives and the like, which may provide non-transitorystorage at any time for software programming.

Software may be communicated through the Internet, a cloud serviceprovider, or other telecommunication networks. For example,communications may enable loading software from one computer orprocessor into another. As used herein, unless restricted tonon-transitory, tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

The foregoing general description is exemplary and explanatory only, andnot restrictive of the disclosure. Other embodiments may be apparent tothose skilled in the art from consideration of the specification andpractice of the invention disclosed herein. It is intended that thespecification and examples be considered as exemplary only.

What is claimed is:
 1. A computer-implemented method for processingelectronic medical images to predict one or more donor recipients for apatient, comprising: receiving a digital medical image of the patient,wherein the patient is in need of a transplant; determining a trainedmachine learning system; providing the digital medical image into thetrained machine learning system, the trained machine learning systemdetermining a patient embedding; determining, using the patientembedding, a subset of donor recipients; and determining based on thesubset of donor recipients a recommendation of optimal donors.
 2. Themethod of claim 1, wherein the transplant is a fecal matter transplant.3. The method of claim 1, wherein the transplant is a liver transplant.4. The method of claim 1, wherein a salient region detection module isapplied to determine a saliency of each region within the receiveddigital medical image, and non-salient image regions are excluded fromprocessing by the trained machine learning system.
 5. The method ofclaim 1 further including: receiving metadata associated with thepatient, the metadata comprising: clinical data, genetic information,microbial composition, and/or life history data; and providing themetadata into the trained machine learning system.
 6. The method ofclaim 1, further including: determining a second trained machinelearning system, the second trained machine learning system beingcapable of determining a dietary, sleep, or exercise suggestion, whereinthe second trained machine learning system receives as input the digitalmedical image of the patient, determines a lifestyle embedding, anddetermines, based on the lifestyle embedding, a dietary, sleep, orexercise suggestion for the patient.
 7. The method of claim 1, furtherincluding: notifying, with a notification, donor collection entities ofa need for donors with a given profile.
 8. The method of claim 7,wherein the notification may include metadata of the patient in need, anindication of the donor profile needed that matches the recipient,and/or a request for the donor collection entities to start collectingdonors that have a similar profile.
 9. A system for processingelectronic medical images to predict one or more donor recipients for apatient, the system comprising: at least one memory storinginstructions; and at least one processor configured to execute theinstructions to perform operations comprising: receiving a digitalmedical image of the patient, wherein the patient is in need of atransplant; determining a trained machine learning system; providing thedigital medical image into the trained machine learning system, thetrained machine learning system determining a patient embedding;determining, using the patient embedding, a subset of donor recipients;and determining based on the subset of donor recipients a recommendationof optimal donors.
 10. The system of claim 9, wherein the transplant isa fecal matter transplant.
 11. The system of claim 9, wherein thetransplant is a liver transplant.
 12. The system of claim 9, wherein asalient region detection module is applied to determine a saliency ofeach region within the received digital medical image, and non-salientimage regions are excluded from processing by the trained machinelearning system.
 13. The system of claim 9 further including: receivingmetadata associated with the patient, the metadata comprising: clinicaldata, genetic information, microbial composition, and/or life historydata; and providing the metadata into the trained machine learningsystem.
 14. The system of claim 9 further including: determining asecond trained machine learning system, the second trained machinelearning system being capable of determining a dietary, sleep, orexercise suggestion, wherein the second trained machine learning systemreceives as input the digital medical image of the patient, determines alifestyle embedding, and determines, based on the lifestyle embedding, adietary, sleep, or exercise suggestion for the patient.
 15. The systemof claim 9 further including: notifying, with a notification, donorcollection entities of a need for donors with a given profile.
 16. Thesystem of claim 15, wherein the notification may include metadata of thepatient in need, an indication of the donor profile needed that matchesthe recipient, and/or a request for the donor collection entities tostart collecting donors that have a similar profile.
 17. Anon-transitory computer-readable medium storing instructions that, whenexecuted by a processor, perform operations processing electronicmedical images to predict one or more donor recipients for a patient,the operations comprising: receiving a digital medical image of thepatient, wherein the patient is in need of a transplant; determining atrained machine learning system; providing the digital medical imageinto the trained machine learning system, the trained machine learningsystem determining a patient embedding; determining, using the patientembedding, a subset of donor recipients; and determining based on thesubset of donor recipients a recommendation of optimal donors.
 18. Thenon-transitory computer-readable medium of claim 17, further including:receiving metadata associated with the patient, the metadata comprising:clinical data, genetic information, microbial composition, and/or lifehistory data; and providing the metadata into the trained machinelearning system.
 19. The non-transitory computer-readable medium ofclaim 17, further including: determining a second trained machinelearning system, the second trained machine learning system beingcapable of determining a dietary, sleep, or exercise suggestion, whereinthe second trained machine learning system receives as input the digitalmedical image of the patient, determines a lifestyle embedding, anddetermines, based on the lifestyle embedding, a dietary, sleep, orexercise suggestion for the patient.
 20. The non-transitorycomputer-readable medium of claim 17, further including: notifying, witha notification, donor collection entities of a need for donors with agiven profile, wherein the notification may include metadata of thepatient in need, an indication of the donor profile needed that matchesthe recipient, and/or a request for the donor collection entities tostart collecting donors that have a similar profile.